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ABSTRACT 
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critiques current training design models and the lack of attention to system 
factors and shows how attention to system factors can be integrated with a 
popular instructional system design model. "Establishing Internal Quality 
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South Carolina Fire Academy during its first 5 months that was conducted to 
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can be made. "Implementing a New Model for on- the- Job Training: Critical 
Success Factors" (Simone J. van Zolingen, Jan N. Streumer, Rolinda de Jong, 
Marcel R. van der Klink) reports on a study of the effectiveness of a new 
instructional model for training desk employees that was developed by Post 
Offices, Inc. in the Netherlands. Two papers contain reference sections, (MN) 
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Leveraging System Factors for Increased Training Effectiveness 



David Ripley 
University of Canterbury 



World wide, training costs are staggering. The level of expenditure begs the question, **How can 
organisations ensure that this money is well spent - that the training is effective?** This conceptual paper 
critiques current training design models, and the lack of attention to system factors. We show how 
attention to system factors can be integrated with a popular instructional system design (ISD) model, and 
raise questions to help future researchers more comprehensively address this issue. 
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We do not yet know what the total global training bill will be for the year 2000, but five years ago estimates were 
that over 50 billion dollars per year were being spent on training in the US alone (Nelson, Whitener, & Philcox, 
1995). In 1998, it was reported that this figure had soared to over 60 billion dollars (Bernstein, 1998). In 1999, we 
were told that Dow Chemical alone had an annual training budget of 80 million dollars (Briody, 1999). One author 
has suggested that global training expenses just for information technology will reach 18.5 billion US dollars in the 
year 2000 (Violino, 1999). 

Considerable time and attention is given to the issue of producing effective training. This is understandable, 
since anecdotal evidence indicates that companies who spend more on training are more profitable (Sorohan, 1995; 
Cole, 1998) and describes some very high returns on training investment (Simington, 1998). However, others claim 
that $80 out of every $100 training dollars is wasted (Maglitta, 1997). If we accept this latter claim as even partially 
correct, this is an issue that should be of significant concern to both scholars and practitioners because it indicates 
much of our training effort is ineffective. We believe one major contributor to ineffective training is lack of 
attention to the system factors that can undo the benefits of otherwise well planned and executed training. This lack 
of attention to system factors is directly contrary to what we know about the link between training effectiveness and 
system factors from the transfer of training literature (Tharenou, 1995; Rouiller & Goldstein, 1993; Tracey, 
Tannenbaum, & Kavanaugh, 1995: Brinkerhoff & Montesino, 1995; Machin and Fogarty, 1997). The literature 
makes it clear that the system in which the training must be applied will significantly impact results on the job. 

It has been said that if we put a good performer in a bad system, the system will win every time. We suggest a 
variation on that theme. "Return a well-trained employee to an unprepared system, and the training dollars may well 
have been wasted." We define the term "system factors," for purposes of this paper, as those things in the work 
environment that influence employees' work-related behaviours, and "the system" as those factors in their entirety. 

The object of training is normally to enhance employee skills and knowledge with the aim of improving 
performance. Therefore, we begin the paper with a discussion of employee performance with particular emphasis 
on the importance of system factors. The design of the training is discussed next. Instructional systems design in 
general is discussed, and the major components of one popular ISD model are reviewed, as a vehicle for pointing out 
how system factors are typically not taken into account in training design. We then review transfer of training 
literature to show that the lack of emphasis on system factors impacts upon the effectiveness of the training 
produced with our normal design approaches. This is followed by a discussion of key system leverage points that 
we believe must be considered in training programme design, and that we believe will lead to increased transfer of 
training and improved performance. We show how these can be integrated with a popular ISD model, and conclude 
the paper with questions for future research that are raised by the integrated model. 



Employee Performance and the Importance of Person and System Factors 

The literature of the past few years indicates emphasis in business organisations on reinvention, reengineering, and 
transformation, all in the pursuit of enhanced employee and organisational performance (Hammer & Champy, 1993; 
Passmore, 1994; Redding & Catalanello, 1994; Runimler & Brache, 1995). 

HRD scholars continue to investigate various aspects of employee performance. Ruona and Lyford-Nojima 
(1997) identified over a score of the authors and researchers who have provided foundational and research work to 
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help us understand major variables that contribute to employee performance such as mission/goal, systems design, 
capacity, motivation, and expertise. 

Both person-specific factors and system-related factors are critical to performance. Carson, Cardy, and 
Dobbins (1991) presented a model that demonstrates this interaction, shown in Figure 1. The authors divided 
employee performance into two categories, job-related behaviours and work outcomes. Their model showed that 
system factors in the work environment impact both behaviours on the job and work outcomes. They believed 
system factors have both a potentially positive influence and the potential to constrain performance. 

This model is very similar to the approach of Gilbert (1996). Gilbert defined performance as a flmction of both 
behaviours and accomplishments. His Behaviour Engineering Model indicates that work environment (system) 
factors as well as individual factors impact employees and their performance. Other authors have recognized that 
system factors as well as person factors can influence performance as well (Bemardin, 1989; Blumberg & Pringle, 
1982; Ilgen and Favero 1985; Olson & Borman, 1989; Peters, O'Connor, & Eulberg, 1985). 

Figure 1. Relationship of Person and System Factors to Two Domains of Performance 




Person Factors. A person-factor perspective of performance (Hellriegal, Slocum, and Woodman, 1995) can be 
shown as Performance = f (ability X motivation). From this perspective, performance comes directly from 
employee behaviours, which in turn are a function of motivation and ability. Effective training is one of the major 
tools for increasing motivation and ability, as shown in figure 2. 

Figure 2. The Impact of Training on Person Factors that Contribute to Performance 




Person-factors have been the subject of considerable study. Text books on organisational behaviour typically 
contain both content and process theories that seek to explain the motivation aspect of the person-factor focus 
(Hellriegal, Slocum, & Woodman, 1995; Ivancevich, & Matteson, 1987; Kreitner & Kinicki, 1998). The "big five" 
personality dimensions have been examined in this regard (Barrick and Mount, 1993; Kreitner and Kinicki, 1998). 
Welboume, Johnson, and Erez (1998) developed a role-based performance scale, using role theory and identity 
theory to develop a measure of performance. Stajkovic and Luthans (1998) have approached individual 
performance fi*om the perspective of social cognitive theory and its derivative, self-efficacy. However, employee 
performance, as Carson, Cardy, and Dobbins (1991) noted, is not just a flmction of the person. Person factors such 
as skill and knowledge are important, certainly, but they are not the whole story. In fact some take the position that 
person factors are a minor factor in determining employee performance. 
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System Factors. Gilbert stressed that work environment issues influence human performance much more 
heavily than individual issues (Gilbert, 1996). Total Quality Management (TQM) theory as espoused by Deming 
(1986) also takes the position that most of what determines human performance is out of the individual employee^ 
hands. System factors that can influence performance have also been studied extensively. Thompson (1993) 
advocated expanding the performance formula to be expressed as performance = f (skill + effort) * (efficacy of 
system being used), and suggested that in order to improve performance with the least investment of resources, the 
investment should be in the system. Employee involvement in planning, overall organisational culture, perceived 
organisational support, and perceived empowerment have all been found to be significantly related to various 
measures of employee performance (Black and Gregersen, 1997; Shore and Wayne, 1993; Fulford and Enz, 1995). 
Arthur (1994) examined steel "niinimills’' and found that those with "commitment" vs. "control" human resource 
systems had higher productivity. These examples all point to system factors exerting an influence on work-related 
behaviours and performance. 

System factors can influence not only the employee work-related behaviours that lead to performance but 
outcomes directly, as shown in figure 3. An obvious example of this latter influence is simply whether or not 
employees have the opportunity to apply the newly learned skills. Skills and knowledge generally have to be 
applied in a specific system - and that system must support the training objectives for those objectives to be realised. 
Otherwise, the training will not be transferred into the workplace, in terms of application on the job. 



Figure 3. A More System-Focused View of Training and Performance 




If system factors are critical to performance, then logically they should be taken into accoimt in designing 
training programmes that are intended to result in increased performance. But are they? We turn now to a 
discussion of instructional systems design to explore this question. 



Instructional Systems Design (ISD) and the Degree of Importance given to System Factors 

The definition of instructional systems design, also known as instructional design (Rothwell and Kazanas, 1992), 
continues to evolve. Richey (1986, p. 9) defined instructional design as ‘the science of creating detailed 
specifications for the development, evaluation, and maintenance of situations which facilitate the learning of both 
large and small units of subject matter.” Gagne', Briggs, and Wagner (1992, p. 20) offered additional useful 
definitions. They consider instructional systems design the systematic process of planning instructional systems and 
instructional development the process of implementing the plans. Together, the two fimctions are components of 
what they term instructional technology. Romiszowski (1981) discussed how the systems approach began to be 
used in training and education in the late 1950s and early 1960s, and notes that all that was new was the systematic 
methodology; the general approach is as old as the scientific method itself. Also, he conceptually positioned 
instructional technology as existing within performance technology, which in turn exists within human resources 
management technology. 

Many ISD models have been proposed. Tennyson and Foshay (1998) summarised ISD models since the 1960s 
and noted, "There are himdreds of specific ISD models" (p. 65). They categorised ISD models into four 
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“generations” based on a common set of system attributes and described the evolution of models from the first to the 
fourth generation. 

One popular design model is the "ADDIE” model, for analysis, design, development, implementation, and 
evaluation, which Schlegel (1995) has referred to as the "generic" model. Some believe ADDIE has outlived its 
usefulness and should make way for more up-to-date models that are less linear and behaviouristic (Gayeski, 1998). 
Tennyson's fourth generation ISD (ISD"^) would be such a model (Tennyson, 1999). The ISD"^ approach is certainly 
more dynamic and interactive and less linear in design ; it can accommodate constructivist as well as behavioural 
theories of learning, and is more inclusive. It is also more situational, in terms of tailoring an approach to fit the 
specific training situation (Tennyson & Foshay, 1998). However, we suggest that while ISD^ models feature major 
changes in the areas Gayeski (1998) discussed, the fiindamental activities of ISD"^ models appear to be the same as 
those in the ADDIE model. That is, we are still concerned with determining what training is needed and 
establishing objectives, designing and developing the training, and implementing and evaluating it. What appears to 
have changed is how effectively we carry out those activities, particularly in terms of flexibility and situational 
responsiveness. Since we are concerned here primarily with what is done during training design — the major 
activities - the ADDIE model is still quite useful for illustration purposes in considering the problem under 
discussion. A simplified version of the model is shown in figure 4. 



Figure 4. The ADDIE Model for Instructional Design (Simplified) 
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Romiszowski (1981) cautioned that any flow chart-type model (such as figure 4, above) could be ineffective and 
even misleading in attempting to present a systems approach. The systems approach to problem solving is heuristic, 
not algorithmic. Analysis, design, and evaluation, rather than being highly sequential, will be occurring at all stages 
of the process. The concerns of Romiszowski and Gayeski not withstanding, ADDIE remains a popular model for a 
basic view of ISD. We acknowledge that its application in practice is more complex than figure 4 might lead one to 
believe, but stand by our position that it represents the major design activities. We will examine each of the model's 
five major phases in terms of the extent to which each phase does or not recognise system factors. 

Analysis. A number of approaches have been advocated for the analysis phase of ISD. Scott and Deadrick 
(1992) advocated the use of nominal group technique at the organisation, operations, and person level. McClelland 
(1993) discussed the use of individual interviews, focus groups, on-site observations, and in general, advocates an 
open systems approach that uses many sources to develop training needs information. Regardless of the approach 
used, the intent is generally to determine the specific skills and knowledge issues that training must address and 
establish goals and objectives based on those issues. Some have broadened that approach. Sleezer (1993) indicated 
that using a systems approach might lead the analyst to recognise that the problems under study have multiple 
causes and solutions. Rossett (1987) noted that the training needs assessment might surface causes for which 
training is not the solution. Front-end analysis (FEA), first popularised by Harless (1973, 1975, 1989), is often used 
at the analysis stage to see if the problem is, in fact, in the system. However, FEA is often concerned with a person 
or system focus rather than a person and system focus. We believe a major concern in analysis should be about 
finding the system intervention(s) that should go with person-factor interventions such as training. 

Design, Development, and Implementation. When the above analysis phase indicates training is the appropriate 
response to the problem, one proceeds with design, development, and implementation phases of ISD. Tennyson and 
Foshay (1998) commented that ISD has become much more dynamic and iterative as it has evolved. They discussed 
how what they describe as the "ID domains" (p. 70) have grown from basic design activities and production of 
materials to include implementation and maintenance. Fundamentally though, ISD still reflects a person-factor 
approach in these phases, concerned with producing programmes that will best enhance the skills and knowledge of 
those who participate in the training. Systematically designed training will take system factors into account, but 
typically will not make changes in the system part of the design. 

Evaluation. The last step in the design process is evaluation and, as with the other phases, there is no shortage 
of literature on the evaluation of training. Dionne (1996) reviewed 20 years of the literature and noted how 
evaluation of training had progressively become of more concern for trainers, managers, and researchers. 
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Kirkpatrick (1996) revisited the 4-level model he first introduced in 1959 (reaction, learning, behaviour, and results) 
and concluded that the content had remained much the same. Kraiger, Ford and Salas (1993) have indicated there 
are no theoretically based models of training evaluation and proposed a classification scheme of learning outcomes 
and associated evaluation measures as a move toward such a model. 

Recognition of system factors is not absent from the evaluation literature. Holton (1996) argued that a proper 
evaluation model should accoimt for the effect of intervening variables that affect outcomes, and indicate causal 
relationships. He included transfer of training conditions as a primary intervening variable. Evaluation, as reflected 
in the ISD models discussed above, generally does not meet Holton's criteria with regard to considering the 
intervening variable of transfer of training conditions. 

Effective Transfer of Training. This lack of attention to system factors in ISD runs directly coimter to what we 
know about the link between training effectiveness and system factors from the transfer of training literature. 
Tharenou (1995) foimd that in an Australian federal agency, supervisor support enhanced training effectiveness. 
Rouiller and Goldstein (1993) concluded that the organisational transfer climate of the work situation affects the 
degree to which learned behaviour will be transferred onto the actual job. Tracey, Tannenbaum, and Kavanaugh 
(1995) foimd that both climate and culture were directly related to post-training behaviours. Brinkerhoff and 
Montesino (1995) found significantly higher training useage among trainees who received two management support 
interventions, a pre-training expectation discussion and an after-training follow on discussion. Machin and Fogarty 
(1997) noted that, "When a lack of support is evident or a lack of opportunity to perform trained tasks exists, these 
factors may inhibit the transfer of training" (p. 102). 

It appears there has been a failure to transfer our knowledge of the impact of these system factors on the 
effectiveness of training into instructional systems design. This problem has been raised in the past. Robinson and 
Robinson (1985) stressed the need to conduct a work environment assessment to imcover barriers to skill transfer in 
the work environment before training begins, and the need to work with line managers to remove these barriers. 
Baldwin and Ford indicated in 1988 that research gaps included the need to test operalisations of training design and 
work environment factors that have an impact on transfer. We were imable to find published evidence to show that 
the work environment factor research gap noted by Baldwin and Ford has been closed. 



Leveraging System Factors for Improving Training Transfer and Performance: An Integrated Model for 
More Effective Training Design 

We propose that three system factor leverage points need to be taken into accoimt in training design models. We 
believe these system factor leverage points are: 

1. The initial training needs assessment that must include an assessment of the system in which the training will be 
applied. In particular, factors that will tend to cancel the impact of the training must be identified and system 
factor objectives defined, along with training objectives. 

2. The design and implementation of system interventions is needed to deal with any factors that have been 
identified as potentially cancelling the impact of training. Intervention implementation should occur prior to or 
concurrent with training. Consideration should also be given to system interventions designed specifically to 
enhance the impact of training. 

3. The evaluation phase must include evaluation of system interventions plus evaluation of system impact on 
training transfer. Any factors in the system that are interfering with training transfer need to be identified as 
quickly as possible, so corrective interventions may be taken before the value of the training is lost. 

System leverage point consideration should be viewed as an integral part of the ISD process, a matter of routine 
in good design. The integrated model below offers one example of how this might be done. The main difference 
between these three points and what we already know about systematically designed training is that we are 
advocating system intervention as a primary component of training design, not just taking the contextual situation 
into account in design. Richey (1992) has noted the need for designers to take context into account, perhaps with 
additional learning activities that provide role models and emphasise the positive consequences of the desired 
behaviour. She also suggested management briefings and involvement in the training, and we know this is 
important from the training transfer literature. While toeing the system into account is commendable and in^ortant, 
we believe there is a need to work more directly with the system itself In other words, go beyond just customising 
the training to fit the system. Certainly training should fit the system, but at the same time we should focus on 
enhancing the system to increase the effectiveness of training. 
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Figure 5 shows how consideration for the system leverage points can be integrated with the normal ADDIE 
design phases. 

Figure 5. The Integrated Design Model Reflecting System Leverage Points 




Evaluation must include both the results of the actual training and the results of 
interventions designed to ensure that the system has a positive impact on training 
effectiveness. Feedback from evaluation may take us back to the development phase, 
the design phase, or even the analysis phase of the instructional design process. 



Conclusion and Questions for Further Research 

At the beginning of this paper we suggested that if we return a well-trained employee to an unprepared system 
our training dollars might well have been wasted. Another option is possible. We can return well-trained 
employees to systems that have been prepared to receive them; systems that are prepared to reinforce that training 
and put it to use. To improve the effectiveness of our training, we need to integrate key system leverage points into 
the instructional design process. Well-designed training can be a major contributor to performance, but if we forget 
the system factors it is unlikely that contribution will be maximised. For that reason we stress the need for 
integrated design models, reflecting system leverage points at key steps of the design process. 

The challenge we suggest faces HRD scholars is to go beyond the simple conceptual model offered here, and 
develop more detailed and comprehensive integrated ISD models. Such models are needed to guide us in designing 
training programmes that take the relevant system factors into account, that prepare the system to accept and 
enhance the training, and that result in increased training transfer and improved employee performance. 

In developing these more comprehensive models, we believe questions for ftirther research will include: 

1 . How can training needs assessment be modified to include an appropriate assessment of the system in which the 
training must be applied, in order to determine both those system factors that could inhibit training transfer and 
those that could enhance it? 

2. What interventions will be most effective in preparing the system to ensure that employees are able to make the 
best use of training when they return to the work setting in which they must apply the training? 

3. What practical evaluation techniques can be developed to enable us to identify the impact of the system on 
attempts by employees to put their training to use, and the impact of the system on performance in ways that are 
unrelated to the training? 
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If we can answer these questions, we can help to ensure that we are producing effective training and that our 
training dollars are being well spent. 
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Establishing Internal Quality Benchmarks for the South Carolina Fire Academy 



John H. Abraham, Jr, 

City of Anderson Fire Department 

Melissa H. Marcus 
Dennis G. Tesolowski 
Clinton H. Isbell 
Clemson University 

When South Carolina dedicated its $23,000,000 fire training facility to replace an existing 20 year old 
complex, a unique opportunity was presented to create an internal performance benchmark that would 
document students 'satisfaction with the quality of training at the beginning of program delivery. The 
new South Carolina Fire Academyi (SCFA) motto, *A Center of Excellence" is a public statement by 
the Academy that is intended to meet the expectations of its customers. 

Keywords: Performance Benchmarking, Evaluation, Performance Measurement 

Benchmarking is recognized as an invaluable tool to create a competitive advantage (Spendolini, 1992). During a 
review of literature, it became apparent that no other statewide training organization had benchmarked its student 
satisfaction through the use of statewide survey methodology. 

All training academies contacted during this study indicated that they rely on individual post-course evaluations 
specific to just-completed training. Therefore, the South Carolina Fire Academy could have a competitive 
advantage over other national training centers if customer (student) satisfaction is assessed using a more 
comprehensive strategy and the results are incorporated in making management decisions about program quality 
(Watson, 1996). Program evaluation activities need to focus on product delivery and mission accomplishment 
instead of theory building. Its essence is to provide feedback leading to a successful outcome defined in practical, 
concrete terms (Isaac and Michael, 1995). A strong customer orientation is critical if successful selling of Academy 
firefighter training is to be based on quality, not price, as perceived by customers/students (Peters and Watemam, 
1991). 

Problem Statement 

The purpose of this research is to focus the resources of the South Carolina Fire Academy and unify the scope of 
operation by establishing an internal benchmark of customer satisfaction that could be used as a baseline for future 
comparisons. 

Based on prior objectives, the South Carolina Fire Academy has developed an instructional delivery system 
through a standardized statewide firefighter training curriculum in order to train a maximum number of career, 
volunteer, and industrial fire service personnel. The goals of the SCFA have been to: 

1. Develop the skills necessary to command and control emergency operations involving fire, rescue, and 
hazardous materials incidents. 

2. Develop managerial and leadership skills for all levels of fire officers. 

3. Develop skills in fire department support functions to include public fire education, fire prevention, 
inspections, and fire investigation (LLR Division of Fire and Life Safety, 1996). 

The staff of the new South Carolina Fire Academy is committed to quality. Each member of the staff and faculty 
of the SCFA is dedicated to providing the highest level of service possible in the delivery and support of training to 
SCFA students and fire service personnel (LLR Division of Fire and Life Safety, 1996). 

Customer satisfaction is listed as one of the four values of the Academy. Therefore, an initial internal 
benchmark of student satisfaction established during the earliest feasible period of operation of the new South 
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Carolina Fire Academy could provide important feedback on how well the Academy meets the stated commitment 
to quality and customer needs. Further commitment is identified by the Executive Committee of the South Carolina 
State Firemen^ Association who voted unanimously to fund the research to establish internal quality benchmarks 
and to join into a partnership with the Academy on the project 



Theoretical Framework and Research Questions 

The research was conducted based on an internal benchmark approach. The benchmarking process was designed to 
assess the quality of the SCFA’s training programs based on input from trainees (students) attending the new South 
Carolina Fire Academy during its first five months of operation. This study was designed to establish an internal 
benchmark of customer satisfaction that could be used as a baseline for future comparisons. Evaluation of fire 
training in the context of expectations is relevant to this study, in that, the student completing training is the output 
of the Academy and the input to the organization represented by the student. Similarly, training as a process must 
respond to feedback in order to adapt to weaknesses and changing needs where feedback is controlled by the 
evaluation criteria chosen by the Academy. 

According to Gay (1992), a strategy of continuous improvement based on benchmarking requires that the 
opportunity to improve be evident to everyone involved with the process being benchmarked. In the case of the new 
South Carolina Fire Academy, a benchmark score of 13 out of a possible 15 represented a relatively high 
benchmark achievement level. However, even if the higher benchmark score is obtained, the potential for 
quantitative and qualitative improvement in customer satisfaction would still be possible. With cooperation from 
the Superintendent of the Academy and identified experts in the field of fire training, the focus of this investigation 
was formalized in the following statistical question. 

Statistical Question: Utilizing the South Carolina Fire Academy Customer Satisfaction Survey, the benchmark 
mean satisfaction score of the new South Carolina Fire Academy is equal to or less than 13 on a scale of 0 to 15. 
Thus, 

Ho: II < 13 
Ha: II > 13 

The null hypothesis would be rejected if the probability of the t-value was less than the one-tailed .05 
probability of the critical value. Additionally, the following subsidiary questions are also addressed in the study. 

Subsidiary Question #7; What, if any, was the difference in student satisfaction between the seven established 
Regional Training Centers of the Academy? 

Subsidiary Question U2: Was there a significant difference in student satisfaction with the Academy training 
administration, curriculum, and instructors? 

Subsidiary Question #5; Did the type of fire department (career, volunteer, combination, or industrial) make a 
significant difference in how firefighters rated the Academy? 

Subsidiary Question #4.* Did the number of courses attended during the study period influence student 
satisfaction with the Academy? 



Methodology 

The methodology of this study included a mail survey instrument, targeting a population of students (who attended 
the new SCFA during the study period), sampling the population, and determining a mean satisfaction score for 
each student based on the data collected with the survey instrument. Subsequently, the average of the students’ 
mean scores became the mean satisfaction score and the measure of the internal benchmark. The mean satisfaction 
score became the dependent variable for the purpose of conducting statistical analyses. Demographic data 
permitted the identification of mean satisfaction scores between regions of South Carolina, type of fire department 
represented by the students, and by number of classes attended by students during the study period. 

Data Collection and Analysis 
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The South Carolina Fire Academy provided a list of 3,482 students who had attended the Academy during the 
study period. The student list was reviewed to prohibit any student from having a higher probability of being 
selected. A random number table was used to select students according to social security number. Once a student 
was selected, the name was removed from the list so that every student had an equal chance of being selected. One 
hundred and fifty randomly selected students were chosen to participate in the study. 

The South Carolina Fire Academy Customer Satisfaction Survey, a cover letter, and self-addressed return 
envelope were mailed to the 150 randomly selected students. A follow-up letter was sent three weeks later to 
students who had not returned their completed survey. Lastly, a third request for response along with a new copy 
of the survey and self-addressed stamped envelope was mailed to individuals who had not responded after six 
weeks. 

The data in Table I shows the total population distribution for the seven Regional Training Centers and those 
students from out-of-state who attended the new South Carolina Fire Academy during the timeframe of the study. 
Sixty (40%) usable surveys were received by the conclusion of the third mailing. Also included in Table I are the 
results of the random sampling and the returns from each region that comprised the data analyzed to determine the 
internal benchmark of customer satisfaction. 

Table I 

Population. Sample, and Returns 



Regions 





1 


2 


3 


4 


5 


6 


7 


Out-of-State 


Total 


Population 


651 


216 


202 


413 


1,098 


618 


232 


52 


3,482 


Sample 


23 


16 


14 


22 


32 


27 


12 


4 


150 


Returns 


7 


10 


6 


5 


10 


13 


4 


• 3 


58* 



* Two additional usable surveys were returned that did not identify a region. 



Reliability and Validity of the Survey Instrument 

The South Carolina Fire Academy Customer Satisfaction Survey was reviewed during its development by a 
panel of experts using a Delphi technique to ensure content validity. Additionally, a group of randomly selected 
South Carolina Fire Academy students pilot-tested the instrument. Post hoc analysis revealed that the questionnaire 
appeared to have criterion-related validity when compared to post-course evaluations collected and analyzed by the 
South Carolina Fire Academy staff independent of this study. The following procedures were implemented in order 
to establish baseline information related to the overall reliability and validity of this instrument. 

Follow-up telephone interviews with five randomly selected individuals, not returning the survey, revealed no 
bias for or against the Academy as a reason for not completing and returning the instrument. This suggests that a 
systematic sampling error was not prevalent. 

A t-test was used to determine if there was a significant response difference between the first one-third (20) of 
the questionnaires returned and the last one-third of the returned questionnaires. At the .05 level of significance, 
there was no significant difference between the earliest and latest returns on any of the 31 questionnaire items used 
to determine the mean satisfaction score for the internal benchmark of student satisfaction. 

Three different methods were utilized to determine the reliability of the instrument. After all returns had been 
accepted, ten respondents were randomly selected to receive a duplicate South Carolina Fire Academy Customer 
Satisfaction Survey for a test-retest reliability comparison. Appropriate documentation was included to explain 
that they had been selected as a test group for determining the reliability of the instrument. They were asked to 
remove themselves from the procedure if they had attended the new South Carolina Fire Academy after the 
timeframe of the study. Eight of the ten re-tests were returned. Utilizing a Pearson Product-Moment Correlation, 
the test-retest reliability was .955. 

In addition to the test-retest reliability analysis, a Cronbach Alpha test was performed to determine the internal 
consistency of the instrument. A Cronbach Alpha of .906 was computed for the South Carolina Fire Academy 
Customer Satisfaction Survey. 
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The third reliability check was completed to verify that the data received on the survey were correctly entered 
into the computer for analysis. To confirm that data were entered correctly, 20% of the qualified returns were 
randomly selected. The audit indicated an estimated error rate of .079%. 



Results and Findings 

The mean satisfaction score was calculated by determining the average of the 31 questionnaire items answered by 
each respondent. Students in the sample population were instructed to respond to each item by placing an “X” on 
a 15 centimeter graphic scale similar to that shown in Figure I. 

Figure 1. Graphic scale Used to Collect Quantitative Data on Survey Instrument 

AGREE DISAGREE 

According to Kubiszyn and Borich (1993), the graphic scale method permitted a common frame of reference for 
comparing all respondents on the same items and to compare item by item. The marks on each of the scales for 
each item were measured using a metric ruler, and the resulting 0 to 15 scores provided a continuous score for each 
item. 

The mean score for each of the 60 qualified respondents was then summed and divided by 60 in order to 
determine the overall mean satisfaction score (13.52 ± .55). The mean and median satisfaction scores, standard 
deviation, and minimum and maximum scores obtained from the South Carolina Fire Academy Customer 
Satisfaction Survey are included in Table II. 

Table II 

Mean Satisfaction Score Descriptive Statistics 



Cases Mean Median Std. Dev. Min. Max. 



60 13.52 14.75 2.15 5.25 15.00 



A one-tailed t-test with the test value of 13 was conducted at the significance level of .05. The results are 
included as Table III. 



Table III 

Customer Satisfaction Benchmark T-Test for Significance Test Value = 13 



Number 








Mean 


of Cases 


Mean 


Std. Dev. 


SE of Mean 


Difference 


60 


13.52 


2.15 


.28 


.52 



t-value=1.86, df=59, one-tail significance=.03 



The one-tailed p-value of .03 is less than the one-tailed probability of .05. Therefore, there was sufficient 
evidence at the .05 level of significance to reject the null hypothesis that the mean satisfaction score was equal to or 
less than the benchmark score of 13. The alternative hypothesis was accepted with the conclusion that the mean 
satisfaction score was greater than 13. It was then concluded that the South Carolina Fire Academy exceeded its 
benchmark target of 13 on a scale of 0 to 15. 

With regard to subsidiary question #1, the importance of the question was to determine whether the delivery of 
SCFA programs was perceived by students as being equal throughout South Carolina. The descriptive statistics for 
the seven different regions are presented in Table IV. 
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Table IV 

Mean Satisfaction Score by Region Descriptive Statistics (N=55) 



Region 


Return 


Mean 


Median 


Std. Dev. 


Min. 


Max. 


Region 1 


7 


13.54 


13.50 


1.62 


11.25 


15.00 


Region 2 


10 


13.78 


14.38 


1.71 


9.75 


15.00 


Region 3 


6 


11.17 


11.25 


3.56 


5.25 


15.00 


Region 4 


5 


11.70 


12.00 


2.51 


8.00 


15.00 


Region 5 


10 


14.13 


14.50 


1.07 


12.25 


15.00 


Region 6 


13 


13.79 


15.00 


2.26 


7.50 


15.00 


Region 7 


4 


14.81 


15.00 


0.38 


14.25 


15.00 



An analysis of variance was used with the mean satisfaction scores for the seven regions as the dependent 
variable and the seven regions as the independent variable. As indicated in Table V, student satisfaction scores 
were significantly different (p=.04) between regions. 



Table V 

Mean Satisfaction Score by Region - Analysis of Variance 



Source 


D.F. 


Sum of 
Squares 


Mean 

Squares 


F 

Ratio 


F 

Prob. 


Between Groups 


6 


61.10 


10.18 


2.41 


.04 


Within Groups 


48 


202.43 


4.22 






Total 


54 


263.53 









To determine which region was significantly different, a Fisher^ Least Significant Difference Test was 
performed (examining all paired comparisons of means) at a significance level of .05. The Fisher^ Test indicated 
that Region 3 was significantly different from Regions 1, 2, 5, 6, and 7 and that Region 4 was significantly 
different from Regions 5 and 7. Further data analysis revealed that the single most significant factor that 
distinguished Regions 3 and 4 from the other regions was the lack of student notification at least 24 hours prior to 
a course being cancelled. 

The second subsidiary question investigated the consistency of training administration, curriculum, and 
instructors throughout the statewide delivery system of the South Carolina Fire Academy. During the development 
phase of the survey instrument, a panel of experts categorized the 3 1 questionnaire items into groups that best 
described the nominal variables of training administration, curriculum, and instructors. The descriptive statistics 
for these variables are reported in Table VI. After the students’ responses were ranked from lowest to highest, the 
Friedman 2-way ANOVA non-parametric test was performed to determine whether there was a statistically 
significant difference between the variables. There was significant evidence (p = .0008) at the .05 level of 
significance to conclude there was a difference between the three variables based on how individual students 
ranked the variables. 

Table VI 

Mean Satisfaction Score by Administration, Curriculum^ and Instructors as Variables - Descriptive Statistics 



Variable 


Returns 


Mean 


Median 


Std. Dev. 


Min. 


Max. 


Administration 


60 


12.97 


13.50 


2.25 


6.07 


15.0 


Curriculum 


60 


12.62 


13.25 


2.56 


6.17 


15.0 


Instructors 


60 


14.02 


14.60 


1.34 


9.18 


15.0 



To determine which of the variables were statistically different from the others, a Wilcoxon Matched-Pairs 
Signed-Ranks test was performed. Based on these results, there was sufficient evidence at the .05 level of 
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significance to conclude that the variable, which described instructors, was ranked significantly higher than the 
administration and curriculum variables. 

The results imply that competent instructors are vital to the success of the South Carolina Fire Academy. 
Competent instructors can deliver high-quality training even when curricula are less than satisfactory and meet the 
needs of students who may not be satisfied with assistance from the SCFA administration. 

The third subsidiary question focused on whether or not satisfaction scores for the public fire departments 
(career, volunteer, and combination) were perceived as being different from that of the private industrial fire 
brigade training. An analysis of variance was performed with the mean satisfaction score used as the dependent 
variable and the type of fire department used as the independent variable with results indicated in Table VIL 

Table VII 

Mean Satisfaction Score bv Type of Fire Department - Descriptive Statistics 



Variable 


Returns ’•' 


Mean 


Median 


Std. Dev. 


Min. 


Max 


Paid Fire Department 


10 


13.65 


15.00 


2.33 


8.00 


15.00 


Volunteer Fire Department 


20 


13.83 


14.38 


1.46 


9.75 


15.00 


Combination Fire Dept 


20 


13.62 


14.63 


2.57 


5.25 


15.00 


Industrial Fire Brigade 


6 


12.92 


14.63 


3.13 


7.50 


15.00 



'•‘Four students were not members of a fire department and were not included in the reported returns. 



The analysis of variance indicated that there was not a significant difference (p=.75, a=.05) between the mean 
satisfaction scores based on the type of fire department. Lower satisfaction scores for industrial firefighters may be 
linked to their broader exposure to other professional organizations. 

The fourth subsidiary question investigated the consistency of the training experience over multiple training 
exposures. A new student may be impressed with new facilities and/or curriculum with the halo effect being 
present during an assessment. More frequent training may result in lower satisfaction after the facility and/or 
curriculum have become routine. 

Table VIII includes the arrangement of the number of classes attended into four discrete categories and the 
descriptive statistics for each. To perform this test an analysis of variance was performed with the mean 
satisfaction score used as the dependent variable and the number of classes attended used as the independent 
variable. 

Table VIII 

Mean Satisfaction Score bv Number of Courses Attended - Descriptive Statistics 



Courses Attended 


Returns 


Mean 


Median 


Std. Dev. 


Min. 


Max 


1-2 


42 


13.62 


15.00 


2.09 


7.50 


15.00 


3-4 


13 


13.14 


13.14 


2.65 


5.25 


15.00 


5-6 


4 


13.31 


13.00 


1.33 


12.25 


15.00 


7 


1 


15.00 











The analysis of variance revealed that the number of classes attended did not make a difference (p=.81, a=.05) 
in how students perceived the quality of their training experience at the new South Carolina Fire Academy. 

Although there was no statistical difference in student satisfaction scores by number of courses attended, the 
data indicated a low negative correlation between these variables (-.02). Therefore based on this data, a slight 
potential exists for students to become less satisfied the more they attend training at the South Carolina Fire 
Academy. 



Conclusions and Recommendations 



O 
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This benchmarking study concluded that the new South Carolina Fire Academy exceeded the established 
internal student satisfaction benchmark target of 13 out of a possible 15 as determined by the 1996 South Carolina 
Fire Academy Customer Satisfaction Survey. The mean satisfaction score (13.52 ± .55) based on a random sample 
of students who attended the Academy during its first five months of operation supports this conclusion. The fact 
that the new South Carolina Fire Academy exceeded its internal benchmark in this assessment should be a 
motivating factor for Academy employees and a promotional asset for recruiting future students. However, the 
range of responses (5.25 to 15.00) does create the awareness of issues related to administration, curriculum, and 
instructors that affect student satisfaction. Subsequently, the Academy has implemented a strategy of continuous 
quality checks based on student feedback and internal management controls. 

The Academy has also implemented programs to obtain consistency between its seven Regional Training 
Centers. The significant difference in satisfaction between Regions 3 and 4 and the remaining five regions was 
reviewed to determine the different business and training practices that contributed to student satisfaction and 
dissatisfaction. 

The staff of the new South Carolina Fire Academy continues to be cognizant of its interactive environment with 
other training agencies, especially in the context of training industrial firefighters. The growth of industrial 
training is essential to the financial business plan of the new SCFA. Therefore, the South Carolina Fire Academy 
has expanded its research of best training processes and is continually seeking to implement those improvements 
where possible. 

The South Carolina Fire Academy, through its professional development training for full-time and part-time 
instructors, reinforces the importance of monitoring student satisfaction. Quality awareness is also communicated 
to students as they attend Academy training by encouraging students to report problems as soon as possible so that 
the training experience is not impeded by an unsatisfactory, but correctable, circumstance. 



Contribution of Knowledge in HRD 

This study, involving the assessment of students who attended the new South Carolina Fire Academy during its 
first five months of operation, established the baseline of student satisfaction against which future comparisons can 
be made. The practice of benchmarking requires a continuous process of evaluation to determine the trend of the 
criteria being evaluated. Now that an internal benchmark of customer satisfaction has been established, 
longitudinal studies can be conducted on a regular basis to monitor adherence to this benchmark standard. 

The use of the South Carolina Fire Academy Customer Satisfaction Survey, or similar instrumentation, should 
be expanded to other statewide fire service training facilities. If other training academies are willing to participate 
in such an assessment, a standardized comparison of fire service training would be possible. 

With the commitment to quality by the South Carolina Fire Academy and the support from the fire service in 
the State, the Academy^ goal of becoming a ‘Center of Excellence” for fire service training will stay in sharp 
focus. 

The process of determining customer satisfaction is transient to other organizations. All organizations that 
provide training services to their internal and external customers must be concerned about quality and the 
perceived-level of customer satisfaction if they are to survive in the 21^* century. 
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Post Offices Inc. in the Netherlands has developed and implemented a new instruction model for the 
training of desk employees. The quality of the new instruction model was assessed by means of the 
evaluation model of Jacobs & Jones for on-the-job training. It is concluded that the implementation of the 
training model has not been completely successful. It is concluded that the attitude of the mentors as well 
as the content of the self study material have to be improved. Besides, the duration of the course has to be 
prolonged. 



Key Words: On-the-Job Training, Transfer of Training, Training Development 

Post Offices Inc. in the Netherlands has been training new employees for more than 25 years. Increased demands for 
quality and a need for more imiform services have led the Human Resources department of Post Offices Inc. to 
develop and implement a new instruction model for the training of desk employees in small, privately-owned post 
offices. It was decided to use (structured) on-the-job training (OJT) because of the many advantages that it offers 
and the appropriateness of the method in this specific case. * 

OJT has well-known advantages, such as; (1) a strong link between training and practice, which has a positive 
impact on trainees’ motivation; (2) skills acquired on the job are learned more effectively (faster and with high 
retention levels); (3) favorable cost-effectiveness; (4) flexibility - OJT is very flexible, can be delivered just in time, 
and can be easily adapted to changes; (5) the transfer problem is minimized, since training is given on the site where 
the employee will work. 

OJT was thought an appropriate method in this special case because Post Offices Inc. achieved a ‘positive score” 
on the selection criteria mentioned by Jacobs & Jones (1995). These criteria include: the nature of the task (training 
is necessary immediately and involves new employees, the tasks to be learned are not very difficult, and the 
consequences of errors can be minimized); available resources (mentors, equipment, tools and data are available); 
constraints in job setting (available training locations, low work distraction); financial considerations (large number 
of trainees); and individual differences (trainee prerequisites and preferences are both unknown). 

Nevertheless, there are also considered to be a number of disadvantages to OJT: (1) supervisors/line managers 
often perceive OJT as an extra burden; (2) the atmosphere on the work site may not be favorable to starting up the 
learning process and keeping it going; (3) a heavy pressure of work can impact negatively on the training process; 
(4) learning materials are often not kept up to date; (5) because of time constraints, little attention is paid to the 
necessary background of the skills and knowledge to be learned; (6) OJT preparation of the trainers who will have to 
deliver OJT is mostly inadequate. 

Post Offices Inc. decided to evaluate the new instruction model twelve months after its implementation. The 
study focused on the following research question: Is the quality of the instruction model considered to be 
satisfactory in terms of instructional materials, learning results, and the practical behavior of employees? 



Theoretical background 

Jacobs & Jones (1995) and Rothwell & Kazanas (1994) state that workplace training can be divided into structured, 
planned on-the-job training, usually referred to as OJT, and unstructured, unplanned on-the-job training. Structured 
on-the-job training is planned instruction occurring on the job during work, centered on the knowledge and skills 
workers should possess in order to perform competently (Rothwell & Kazanas, 1994). Structured OJT can be 
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defined as an organized, structured, intentional form of training that contains well-directed pedagogical 
interventions, in which the work site functions as a place for learning (De Jong, 1998). An additional characteristic 
of OJT, according to Van der Klink (1998), is that on-the-job training involves intentional learning and that, as a 
consequence, a (formal) training arrangement is required that includes the intended training objectives. Structured 
OJT may be delivered by a supervisor, an experienced co-worker, a subordinate, or a job coach from outside the 
organization, or it may be self-directed and thus overseen by the employees themselves. It usually involves one-on- 
one instruction. In developing structured OJT there is often first an extensive job analysis, followed by a step-by- 
step instruction method. Unstructured, unplanned OJT also occurs at the work site. This involves informal methods 
of learning (non- intentional learning, learning as a by-product of the daily scheduled tasks). A plan or training 
arrangement is non-existent and employees learn by, for instance, imitating experienced workers. 

One interesting new phenomenon is the introduction of concepts such as the ‘learning organizat ion”, ‘lifelong 
learning”, and ‘brganizational learning”. These concepts are based on the idea that organizations cannot build on the 
individual learning of their employees, but that this learning needs to be shared and acted upon in such a way that 
the organization performs outstandingly in an increasingly competitive environment (Senge, 1990). Organizations 
should operate a continuous, organizational learning cycle (Nonaka, 1991), where knowledge is created, captured, 
shared and implemented, preferably by teams of workers. In learning organizations the role of HRD professional is 
changing from training specialist to performance improver (Robinson & Robinson, 1998), focusing on the creation 
of opportunities, learning environments, in which individuals and teams can learn and share each others knowledge. 
These developments shed a different light on on-the-job training. On-the-job training should be read as on-the-job 
learning, which means that the HRD professional has to create situations in the worlq)lace in which employees are 
invited and encouraged to learn (together). It should also be emphasized that the HRD professional has to take into 
account a shift in learning content: besides instrumental, job-specific knowledge and skills, growing importance is 
being attached to self- regulating and team-regulating competencies (meta-cognitive competencies: e.g. planning, 
monitoring, and assessment activities; socio-communicative and socio-normative skills). 



Underlying reasons prompting the study 

A number of reasons were observed that all pointed to the necessity for implementing a new instructional model. In 
practice, the variation relating to the execution of the training program for desk employees could be considered 
large. Many assessments pointed out that the length of the training program and the application of the training 
materials and tests depended very much on the local circumstances of the post offices and the preferences of the 
mentors. Besides, mentors had often not been trained for this work and were doing it part-time. The introduction of 
the training model was meant to achieve a minimum level of quality. For these reasons it was decided to use full- 
time mentors, starting from the introduction of the new training model. 

The new training model has the characteristics of structured OJT. The model is based on extensive job analysis, the 
learning process is planned, the amount of available training time is restricted, and the results are measured (see 
figure 1). 



Fig 1: The four characteristics of the new instruction model 



1 . The length of the training is 4 weeks. 

2. The training consists of a practical and a theoretical part. The practical part comprises learning on the job by 

selling the ‘products” of Post Offices Inc. to clients at the counter (4 hours a day) under the supervision of a 
mentor. The theoretical part consists of learning the theory by self-study (4 hours a day), with the mentor being 
at the new employee^ disposal. 

3. The instruction material for self-study is provided by the HRD department of Post Offices Inc. Two books and a 

COO-package have to be used during self-study. 

4. To measure the progress of new employees, they must be tested to assess their behavior and knowledge of 

services. 



Evaluation of the training program 
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Kirkpatrick (1979, 1994) developed a model to evaluate training efforts. He identified four levels of evaluation and 
arranged them hierarchically from the least to the most difficult. The lowest and easiest level concerns evaluation in 
terms of learner reactions: did the learner like the training? The second level of evaluation is learning: what was 
learned from the training? The third level is behavior: how much did learners change their behavior as a result of the 
training? The last and fourth level is results: how much organizational improvement resulted from the leamerh 
behavioral change? 

This model has been criticized by several authors. Holton (1996) argues that Kirkpatnckh model is unjustly 
acknowledged by many HRD professionals as the standard in the field. The model is not based on profound 
empirical evidence and is considered to be incomplete. Several authors suggest adding an additional level to 
measure more specifically return on investment (see Phillips, 1995). At best, the model could be labeled as an 
(incomplete) taxonomy, that is to say, a framework not including “. .. all constructs and variables underlying the 
phenomena of interest, thus making validation impossible” ( Holton, 1996, 6). This also suggests that the implied 
‘bausal relations”between the levels of Kirkpatrickh model are not based on empirical evidence, so, to be clear, are 
‘Immistakably non-existent”. 

This does not mean, however, that the model is completely unfit for our purpose. Elements of Kirkpatrickh 
model were included in the evaluation model that Jacobs & Jones (1995) developed to measure the results of 
structured OJT. Jacobs & Jones’ model has all the characteristics of a systems model, and consists of four 
components: training outputs, training processes, training inputs and organizational context. The training output 
questions relate to whether the training objectives were met. (e.g. Were the training objectives achieved? What were 
the effects on job and organizational performance? Were training outcomes consistent with the trainee^ 
development needs?). The training process questions focus on the behaviors of the mentor and the trainee during the 
training (e.g. Did the mentor use the model as intended? Did the mentor use effective commxmication skills? Did the 
trainee attend to the mentor?). The training input questions focus on the system components present at the time of 
training: l earnin g tasks, training design and training module, training location, and trainee. Questions relating to this 
are: Was the tr ainin g module accurate and complete and appropriately formatted? Did the trainee have the 
prerequisites needed for training? The organizational questions address the support for the trainee within the context, 
such as the role of supervisors and colleagues, constraints caused by production process pressures, time constraints, 
and the quality of the tutorh training (employees in the role of mentor). The questions included the following: Did 
management provide sufficient resources to support OJT? Can OJT occur within the constraints of the production or 
service delivery schedule? 



Methodology 

This section describes the methodology of the study. The following topics come up for discussion consecutively: the 
research questions, the subjects and settings, and the data collection procedure. 

Research questions 

The overall research question (Can the quality of the new instruction model be considered to be satisfactory?) 
was divided into sub-questions. These sub-questions were clustered according to the four components of the 
evaluation model of Jacobs & Jones (1995): training outputs, training processes, training inputs and organizational 
context. 

Training outputs 

- Did the trainee h behavior in dealing with clients at the desk meet expectations? 

- Did the knowledge achieved by the trainees readi a predetermined level? 

Training processes 

- Did the mentor implement the new instruction model completely, as intended by the designers? 

- Did the mentor apply all four characteristics of the new instruction model, such as a training duration of four 
weeks, 4 hours’ on-the-job training, alternated with 4 hours’ self-study, application of the instructional materials 
provided by the HRD department, testing the learning results of the trainees four times in the program? 

Training inputs 

- Were the trainees’e ntrance level and prerequisites, the mentorh method of instruction, and the complexity of the 
tr ainin g material taken into accoxmt during the development of the new instruction model? 

- Was the quality of the training materials (the textbooks, the COO-package, the learning guide) judged 
satisfactory? 

- Did the different participants/stakeholders have a positive attitude towards the new instruction model? 

- Did the beliefs and the behavior of the mentor correspond with the vision defined by the HRD department? 
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Organizational context 

- Did the management of Post Offices Inc. provide the necessary resources to support the implementation of the 
new instruction model? 

- Was there a conflict between being a productive trainee (selling products at the counter) and, at the same time, 
being expected to learn new behavior through OJT? 

- Did the mentor have enough time available to train the trainees? 

Subjects and settings 

Two studies were in fact carried out: a small-scale survey and a case study. The following groups of persons 
participated in the survey: instructional designers (N=2), mentors (N=33), regional managers (N=7), employers (in 
this study: self-employed shopkeepers) (N=3) and trainees (N=18). In addition to the survey, case studies were 
carried out in four situations to obtain a deeper insight into the training process and the factors influencing it. In the 
case studies the number of participants were: mentors (N=4), trainees (N=5), employers (N=3), regional managers 
(N=7) and training designers (N=2). 

Data collection procedure 

Data collection took place by means of a survey of all participating post offices and four selected case studies. 
Figure 2 mentions all the sources and instruments used. 

Although the collected data were processed quantitatively per question/item in each instrument, in this paper the 
results are discussed qualitatively at an aggregated level: all collected data (per research question) are taken together, 
which implies that all the available source material contributed to the composition of the answers. 



Figure 2: Instruments for data collection and data sources used to answer the research questions 



Mentors 

Data from mentors were collected by means of a questionnaire and, if they were participating in the case studies as 
well, a semi-structured interview and a log. 

Employers 

Employers were asked to fill out the standard evaluation form of the HRD department of Post Offices Inc. and, if 
they were also participating in the case study, an interview was held. 

Regional managers 

Regional managers filled out the standard evaluation form and, if also involved in the case studies, a semi- 
structured interview was held. 

Designers 

Designers were questioned by means of a semi-structured interview. 

Trainees 

Finally, the trainees filled out a questionnaire and the standard evaluation form, and had to pass four written tests 
(one each week of the training program). 



Results 

In this section the results are discussed in the order in which they were used by Jacobs & Jones (1995). 

Training outputs 

- Did the trainee^ behavior in dealing with clients at the desk meet expectations? 

The employers who were involved in the case study were on the whole satisfied with the way the trainees 
performed their tasks at the desk. Since the WAVE-test - an observation set used by the mentors to assess the 
trainee^ behavior in dealing with clients at the desk - was applied by only 15 of the 30 mentors, the findings are, 
to a certain extent, subjective. 

- Did the knowledge of the new employees reach a predetermined level? 

The trainees were tested at the end of each week of the training program. Two tests were administered, on 
different subject matter: the mail traffic test and the post bank test. The scores were expressed in percentages. The 
assumption was that the trainees’ score would increase as the training program progressed. At the end of week 1 a 
score of 35% was expected, at the end of week 2, a score of 50%, at the end of week 3, a score of 65%, and at the 
end of week 4, a score of 80%. Data were collected from 10 mentors. Nine mentors stated that the score of 80 % had 
not been reached at the end of the program. They explained that the overall criterion was too high and the subject 
matter too broad. These results were confirmed when the trainees’test results were compared (collected at different 
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moments: see test 1 etc. in table 1). It appeared that the criteria for test 1 and test 2 - 35% and 50% respectively - 
were too low and the criterion for the last test (test 4) was too high. Trainees scored on average 59%, respectively 
55% in test 1, 64% and 60% in test 2, 67% and 68% in test 3 and, finally, 72% and 74% in test 4. 



Table 1: Results of the theory tests taken by the new employees 



N=39 


Mean 


SD 


Mail traffic 
test 1 


59 


11 


test 2 


64 


9 


rest 3 


67 


7 


test 4 


72 


10 



Postbank 



test 1 


55 


7 


test 2 


60 


7 


rest 3 


68 


11 


test 4 


74 


9 



Training processes 

- Did the mentor fully implement the new instruction model, as intended by the designers? 

- Did the mentor apply all the four characteristics of the new instruction mode 1 in the program, such as a training 
duration of four weeks, 4 hours’ on-the-job training, alternated with 4 hours’ self-study, application of the 
instructional materials provided by the HRD department, testing the learning results of the trainees four times? 

The length of the training program was considered the minimum to achieve the training objectives. Since it is 

considered to be very difficult to start work as a trainee in a post office that has been set up very recently or - even 
worse - that is being set up at the time of the trainee^ arrival, many mentors send trainees for their first, and 
sometimes second, week to a post office that is operating well. It should be understood that this problem occurs 
frequently as a consequence of the implementation of a new policy of Post Offices Inc. to close their own post 
offices and set up new, small post offices in existing, privately-owned (book) stores. 

A considerable number of mentors were of the opinion that the written self-study material and, to a lesser extent, 
the COO-package was too extensive; this resulted in not all the material being studied by the trainees. Only 20 
mentors used the week planning and the checklist of activities from the mentors’ guide; only 12 administered the 
learning (theory) tests, due to a lack of time. The WAVE-test - an observation set the mentor used to judge the 
trainee^ behavior in dealing with clients at the desk - was used by only 15 of the 33 mentors, because the mentors 
did not value it. (Some think that this practical test cannot be scored objectively, while others are of the opinion that 
the practical test should be administered after the trainees have learned the necessary (prerequisite) knowledge.) 

The designers took into account the trainees’ level and their prere quisites. The mentors were selected after the 
training program had been developed, which meant that they did not participate in the design and development 
process. This was considered to be a handicap, because the mentors were not always aware of the paradigm that is 
basic to the training program. The designers tried to reduce the complexity of the program content by using the 
modularization principle and by applying clear sequencing principles. The training program seemed to have been 
prepared for ‘hormal”post offices, and not to have taken into account the specific situation in the new, privately- 
owned post offices. 

Training inputs 

- Were the entrance level and prerequisites of the trainees, the mentor^ method of instruction, and the complexity 
of the training material taken into account during the development of the new instruction model? 

It became clear from the interviews with the designers of the training program that the (educational) entrance 
level of the trainees had not been taken into account during the development process. There had also been no 
communication with the trainees during the development process. Trainees were only expected to give feedback 
on an evaluation sheet at the end of the training program. 
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The mentors were recruited after the development of the new instruction model and were told how to instruct. 
This meant that the mentors did not have the opportunity to deliver input to the designers. From an innovative 
perspective, this was not very sensible. 

- Was the quality of the training materials (the textbooks, the COO-package, the learning guide) judged 
satisfactory? 

Eighteen of the mentors stated that the theoretical part of the training program did not prepare the trainees 
adequately for their work at the desk. Too much information was presented. Essential information only should be 
presented briefly so that everything can be worked through in the time available. The mentors would like to add 
job aids; the district managers were of the same opinion. 

The mentors judged all the 41 modules of the training program. They rated which modules were most important 
for job performance, which were very difficult, which very motivating, which too extensive and, finally, which 
took more or less time than expected. Nineteen mentors did not apply the sequence of 41 modules, as 
recommended in the program, because this did not link up with the trainees’ practical work as expressed in 
activities at the desk. Only eight of the mentors approved the recommended sequence of modules. 

- Did the different participants/stakeholders have a positive attitude towards the new instruction model? 

The mentors, the district managers and the developers of the instruction material criticized the training program 
on the same points: there was not enough time available and there was too much subject matter. Nevertheless, all 
respondents commented positively on the daily alternation between theory and practice, which was based on the 
central paradigm underlying the new training program. 

- Did the beliefs and the behavior of the mentor correspond with the vision defined by the HRD department? 
Mentors ‘)>lay” four roles: the mentor as mirror, instructor, supporter, or confidential agent. Most (28) mentors 
saw themselves as an instructor. In second place, 25 mentors saw themselves as a supporter; the role of 
confidential agent was mentioned by 22 mentors, and that of mirror by 21 mentors. 

Organizational context 

- Did the management of Post Offices Inc. provide the necessary resources to support the implementation of the 
new instruction model? 

One of these resources is an adequately trained mentor. The mentors mentioned that the training content was not 
difficult for them to teach, although a few mentioned that the pressure of work was too heavy. A quiet place to 
study the self-study books is another important condition, frequently missing. The mentors indicated that 27 post 
offices did not have such a quiet place to study. The mentors’ solution was to let the trainees take their self-study 
books home. 

- Was there a conflict between being a productive trainee (selling products at the counter) and, at the same time, 
being expected to learn new behavior through OJT? 

The training program was designed so that clients would be inconvenienced as little as possible. The mentors 
indicated that clients were aware of the fact that training was going on, but that they reacted very positively and 
patiently; only a few clients complained that they had to wait too long. 

- Did the mentor have enough time available to train the trainees? 

The mentors and the district managers indicated that the duration of the whole course was too short and should 
be extended to 6 weeks. 



Conclusion and discussion 

This study shows some of the typical difficulties attached to putting OJT into practice. The overall conclusion is that 
the implementation of the model did not meet all expectations. Taking the data into account, it is reasonable to 
assume that successful implementation of OJT lies in the variety of the products and services of the particular post 
office (a large post office has a broader range of products and services than a small one). The new, small private post 
offices offer, in theory, the same products and services as the large ones that are employed by Post Offices Inc., but, 
in practice, they offer only a limited number of services. The training model assumes that all services have to be 
learned during self-study. However, it is likely that mentors advise trainees to study only those modules that cover 
the services of their own private post office. OJT is only effective if what is learned can be put into practice, and this 
might explain the findings regarding the trainees’ scores on the learning tests. 

A second factor influencing successful implementation concerns the mentors’performance. Mentors are expected to 
serve as a behavioral model, to provide feedback, arrange an adequate environment for self-study, motivate trainees 
for self-study and evaluate trainees’progress on a regular basis. When mentors are not fully convinced of the quality 




23-3 



24 



of the new instructional model, the implementation will not be successful. The following provide some explanation 
of the fact that the mentors did not work according to the guidelines of the new instructional model. 

First of all, the recruitment of mentors and the development of the new instructional model were two different 
projects. Although all the mentors were recruited from a group of experienced desk clerks within Post Offices Inc, 
their experience was not included during the design of the new instructional model. There was thus no opportunity 
to use the feedback of highly skilled desk clerks in the development of the model. 

A second explanation might be that there was only a short, two-day training program scheduled for the mentors. 
There was not enough time to explain the theoretical background of the model, to convince mentors of its quality, to 
discuss the application of the model in practice, or to teach mentors how to cope with problems (such as the lack of 
sufficient time for trainees’ self study). The preparation of the mentors for their mentoring activities was 
insufficient. 
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